{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Xinference is a powerful tool that supports function calling and is fully compatible with OpenAI's Tool Call API. It allows you to seamlessly integrate and utilize the functionality of OpenAI's tools within your own projects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparation\n",
    "\n",
    "First, you need to install Xinference:\n",
    "```shell\n",
    "pip install xinference\n",
    "```\n",
    "\n",
    "Currently, ChatGLM3 / qwen-chat and gorilla-openfunctions-v1 models are available for tool calls. Here, we install a chatglm cpp for fast inference on Mac. If you are not using a Mac, please follow the install instruction: https://github.com/li-plus/chatglm.cpp#python-binding\n",
    "\n",
    "```shell\n",
    "CMAKE_ARGS=\"-DGGML_METAL=ON\" pip install -U chatglm-cpp --no-cache-dir\n",
    "```\n",
    "\n",
    "Then, start the Xinference server by the following command:\n",
    "```shell\n",
    "xinference\n",
    "```\n",
    "\n",
    "The Xinference server will be started:\n",
    "\n",
    "```shell\n",
    "2023-11-02 16:04:55,278 xinference   38878 INFO     Xinference successfully started. Endpoint: http://127.0.0.1:9997\n",
    "2023-11-02 16:04:55,280 xinference.core.supervisor 38878 INFO     Worker 127.0.0.1:32187 has been added successfully\n",
    "2023-11-02 16:04:55,281 xinference.deploy.worker 38878 INFO     Xinference worker successfully started.\n",
    "```\n",
    "\n",
    "Finally, we launch a ChatGLM3 model for tool calls.\n",
    "```shell\n",
    "xinference launch -u my_tool_model -n chatglm3 -f ggmlv3 -q q4_0\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Function calling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an exciting game that showcases the function calling feature. Our objective is to call the AI and obtain a set of lottery numbers to determine if we are lucky enough to win the lottery. To get started, we first need to define a simple function for generating lottery numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "list_red = list(range(1, 34))\n",
    "list_blue = list(range(1, 17))\n",
    "\n",
    "\n",
    "def get_lucky_lottery(num):\n",
    "    total = []\n",
    "    for _ in range(num):\n",
    "        res_red = random.sample(list_red, 6)\n",
    "        res_blue = random.sample(list_blue, 1)\n",
    "        total.append(res_red + res_blue)\n",
    "    return total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we describe two functions as a dict, more details refers to https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = [\n",
    "    {\n",
    "        \"type\": \"function\",\n",
    "        \"function\": {\n",
    "            \"name\": \"get_lucky_lottery\",\n",
    "            \"description\": \"生成双色球彩票号码\",\n",
    "            \"parameters\": {\n",
    "                \"type\": \"int\",\n",
    "                \"properties\": {\"num\": {\"description\": \"生成的彩票号码组数\"}},\n",
    "                \"required\": [\"num\"],\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"function\",\n",
    "        \"function\": {\n",
    "            \"name\": \"lottery_draw\",\n",
    "            \"description\": \"开奖双色球\",\n",
    "            \"parameters\": {\n",
    "                \"type\": \"object\",\n",
    "                \"properties\": {},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to call above two functions according to the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import functools\n",
    "import json\n",
    "\n",
    "FUNCTION_TABLE = {\n",
    "    \"get_lucky_lottery\": get_lucky_lottery,\n",
    "    \"lottery_draw\": functools.partial(get_lucky_lottery, 1),\n",
    "}\n",
    "\n",
    "\n",
    "def handle_response(completion):\n",
    "    assert \"tool_calls\" == completion.choices[0].finish_reason\n",
    "    func_name = completion.choices[0].message.tool_calls[0].function.name\n",
    "    func_args = completion.choices[0].message.tool_calls[0].function.arguments\n",
    "    func_kwargs = json.loads(func_args)\n",
    "    return FUNCTION_TABLE.get(func_name)(**func_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's ask the LLM to generate 5 group lottery numbers and check if we win the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "client = openai.Client(api_key=\"not empty\", base_url=f\"http://127.0.0.1:9997/v1\")\n",
    "completion = client.chat.completions.create(\n",
    "    model=\"my_tool_model\",\n",
    "    messages=[{\"role\": \"user\", \"content\": \"帮我生成5组双色球号码\"}],\n",
    "    tools=tools,\n",
    ")\n",
    "print(f\"Lottery numbers: {handle_response(completion)}\")\n",
    "completion = client.chat.completions.create(\n",
    "    model=\"my_tool_model\", messages=[{\"role\": \"user\", \"content\": \"开奖\"}], tools=tools\n",
    ")\n",
    "print(f\"Lottery draw: {handle_response(completion)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No lucky, I lose the game.\n",
    "\n",
    "```\n",
    "Lottery numbers: [[30, 9, 22, 26, 17, 27, 14], [8, 6, 27, 30, 21, 20, 13], [4, 33, 9, 32, 27, 22, 14], [19, 1, 30, 4, 28, 13, 7], [16, 7, 23, 17, 8, 30, 12]]\n",
    "Lottery draw: [[29, 10, 27, 30, 15, 6, 1]]\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
